2,160 research outputs found
Optimal Complexity in Non-Convex Decentralized Learning over Time-Varying Networks
Decentralized optimization with time-varying networks is an emerging paradigm
in machine learning. It saves remarkable communication overhead in large-scale
deep training and is more robust in wireless scenarios especially when nodes
are moving. Federated learning can also be regarded as decentralized
optimization with time-varying communication patterns alternating between
global averaging and local updates.
While numerous studies exist to clarify its theoretical limits and develop
efficient algorithms, it remains unclear what the optimal complexity is for
non-convex decentralized stochastic optimization over time-varying networks.
The main difficulties lie in how to gauge the effectiveness when transmitting
messages between two nodes via time-varying communications, and how to
establish the lower bound when the network size is fixed (which is a
prerequisite in stochastic optimization). This paper resolves these challenges
and establish the first lower bound complexity. We also develop a new
decentralized algorithm to nearly attain the lower bound, showing the tightness
of the lower bound and the optimality of our algorithm.Comment: Accepted by 14th Annual Workshop on Optimization for Machine
Learning. arXiv admin note: text overlap with arXiv:2210.0786
Momentum Benefits Non-IID Federated Learning Simply and Provably
Federated learning is a powerful paradigm for large-scale machine learning,
but it faces significant challenges due to unreliable network connections, slow
communication, and substantial data heterogeneity across clients. FedAvg and
SCAFFOLD are two fundamental algorithms to address these challenges. In
particular, FedAvg employs multiple local updates before communicating with a
central server, while SCAFFOLD maintains a control variable on each client to
compensate for "client drift" in its local updates. Various methods have been
proposed in literature to enhance the convergence of these two algorithms, but
they either make impractical adjustments to algorithmic structure, or rely on
the assumption of bounded data heterogeneity.
This paper explores the utilization of momentum to enhance the performance of
FedAvg and SCAFFOLD. When all clients participate in the training process, we
demonstrate that incorporating momentum allows FedAvg to converge without
relying on the assumption of bounded data heterogeneity even using a constant
local learning rate. This is a novel result since existing analyses for FedAvg
require bounded data heterogeneity even with diminishing local learning rates.
In the case of partial client participation, we show that momentum enables
SCAFFOLD to converge provably faster without imposing any additional
assumptions. Furthermore, we use momentum to develop new variance-reduced
extensions of FedAvg and SCAFFOLD, which exhibit state-of-the-art convergence
rates. Our experimental results support all theoretical findings
Optimization of ultrasonic-assisted extraction procedure of capsaicinoids from Chili peppers using orthogonal array experimental design
In this present study, the optimal conditions of ultrasonic-assisted extraction (UAE) of capsaicinoids from hot Chili peppers were determined for large scale preparation. First, single factor experiments were performed to optimize the extraction procedure of capsaicinoids, and initial optimized results were: ratio of solvent to mass of 6 to 10 ml/g, extraction temperature of 25 to 35°C, and extraction time of 0 to 30 min. Then, an orthogonal array experimental design (L9(34)) was used to further optimize the extraction procedure. The results of F-test and P-value indicated that the effect order on extraction yield of capsaicinoids from high to low was ratio of solvent to mass, extraction time, and extraction temperature. The maximum extraction yield of capsaicinoids was obtained at ratio of solvent to mass of 10 ml/g, extraction time of 40 min, and extraction temperature of 25°C. Under these conditions, the extraction yields of capsaicinoids were 2.35 ± 0.042 and 3.92 ± 0.089 mg/g for conventional and UAE methods, respectively.Keywords: Chili pepper, ultrasonic-assisted extraction, capsaicinoids, orthogonal array design, single factor experiment
- …